{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.9"
    },
    "colab": {
      "name": "HandlersTimeProfiler_MNIST.ipynb",
      "provenance": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_1FzCFk3a1S3"
      },
      "source": [
        "# Profiling MNIST example with 3-conv. layer network\n",
        "\n",
        "This example demonstrates the usage of `HandlersTimeProfiler`. The example uses MNIST dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BniLCaIAa1S9"
      },
      "source": [
        "## Install requirements\n",
        "\n",
        "We assume that `torch` and `ignite` (nightly) are already installed. We can install it using `pip`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "52PH_jwua1S_"
      },
      "source": [
        "!pip install --upgrade --pre pytorch-ignite"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AgTadgBza1TA"
      },
      "source": [
        "## Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "pycharm": {
          "is_executing": false
        },
        "id": "Y0sJP9iFa1TB"
      },
      "source": [
        "%matplotlib inline\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "import torchvision.transforms as transforms\n",
        "from torch.utils.data import DataLoader\n",
        "from torchvision.datasets import MNIST\n",
        "\n",
        "# A hack to fix the horizontal spill in large output\n",
        "# ref: https://stackoverflow.com/a/59058418/6574605\n",
        "from IPython.core.display import HTML\n",
        "display(HTML(\"<style>pre { white-space: pre !important; }</style>\"))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iK_9cOP6a1TI"
      },
      "source": [
        "from ignite.engine import Events, create_supervised_trainer, create_supervised_evaluator\n",
        "from ignite.metrics import Loss, Accuracy\n",
        "from ignite.handlers import ProgressBar, HandlersTimeProfiler"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5gzxqMMTa1TE"
      },
      "source": [
        "## Loading MNIST"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ChcbbEo_a1TF"
      },
      "source": [
        "mnist_pwd = \"data\"\n",
        "batch_size= 256"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YcV6pX-7a1TG"
      },
      "source": [
        "transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])\n",
        "\n",
        "trainset = MNIST(mnist_pwd, train=True, download=True, transform=transform)\n",
        "trainloader = DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=4)\n",
        "\n",
        "testset = MNIST(mnist_pwd, train=False, download=True, transform=transform)\n",
        "testloader = DataLoader(testset, batch_size=batch_size * 2, shuffle=False, num_workers=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eN7ewRAza1TG"
      },
      "source": [
        "## Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "RuawaUVva1TH"
      },
      "source": [
        "class Net(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Net, self).__init__()\n",
        "        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)\n",
        "        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)\n",
        "        self.conv2_drop = nn.Dropout2d()\n",
        "        self.fc1 = nn.Linear(320, 50)\n",
        "        self.fc2 = nn.Linear(50, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = F.relu(F.max_pool2d(self.conv1(x), 2))\n",
        "        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))\n",
        "        x = x.view(-1, 320)\n",
        "        x = F.relu(self.fc1(x))\n",
        "        x = F.dropout(x, training=self.training)\n",
        "        x = self.fc2(x)\n",
        "        return F.log_softmax(x, dim=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0BFfdk5ka1TJ"
      },
      "source": [
        "## Training Loss"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nXkNMyJua1TK"
      },
      "source": [
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "criterion = nn.NLLLoss()\n",
        "model = Net()\n",
        "model.to(device)  # Move model before creating optimizer\n",
        "optimizer = optim.SGD(model.parameters(), lr=3e-4, momentum=0.9)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dU37erdra1TK"
      },
      "source": [
        "trainer = create_supervised_trainer(model, optimizer, criterion, device=device)\n",
        "evaluator = create_supervised_evaluator(model, metrics={\"acc\": Accuracy(), \"loss\": Loss(nn.NLLLoss())}, device=device)\n",
        "\n",
        "# Attach handlers profiler\n",
        "profiler = HandlersTimeProfiler()\n",
        "profiler.attach(trainer)\n",
        "\n",
        "# Init and attach progressbar\n",
        "pbar = ProgressBar(persist=True)\n",
        "pbar.attach(trainer, metric_names=\"all\")\n",
        "\n",
        "# Evaluate on each epoch using event handler\n",
        "@trainer.on(Events.EPOCH_COMPLETED)\n",
        "def log_validation_results(engine):\n",
        "    evaluator.run(testloader)\n",
        "    metrics = evaluator.state.metrics\n",
        "    avg_accuracy = metrics[\"acc\"]\n",
        "    avg_nll = metrics[\"loss\"]\n",
        "    pbar.log_message(\n",
        "        \"Validation Results - Epoch: {}  Avg accuracy: {:.2f} Avg loss: {:.2f}\".format(\n",
        "            engine.state.epoch, avg_accuracy, avg_nll\n",
        "        )\n",
        "    )\n",
        "\n",
        "    pbar.n = pbar.last_print_n = 0\n",
        "\n",
        "trainer.run(trainloader, max_epochs=10)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T_AIlhvqa1TK"
      },
      "source": [
        "We can see the summary of the profiling results from our run using the `get_results()` method of the profiler as shown below. The output shows total, average and other details of execution time for each handler attached."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S8J8iWyUa1TL"
      },
      "source": [
        "profiler.print_results(profiler.get_results())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F0yJpyC7a1TM"
      },
      "source": [
        "Profiling results can be exported to a CSV file by using the `write_results()` method of profiler."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_6a0t-Xha1TM"
      },
      "source": [
        "profiler.write_results(\"./results.csv\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-hG8hYQba1TM"
      },
      "source": [
        "Following code shows the preview of few rows of the CSV. Each handler has its separate column and the numbers of rows for each column will be equal to the number of times the handler invoked."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9SraFwsDa1TM"
      },
      "source": [
        "import pandas as pd"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f4ZiVInXa1TO"
      },
      "source": [
        "results = pd.read_csv(\"./results.csv\")\n",
        "results.head()"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}
